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(54) Adaptive image coding 

(57) A method and system for transferring data files 
from a server entity, such as a server computer (102), to 
a client entity, such as a client process running on a 
remote client computer (104, 106) stores the data files 
in a frequency domain (908) form by the server entity 
(102). The client entity (104, 106) can specify certain 
characteristics for the transfer that may be represented 
as one or more parameters. Parameters may include a 
compression ratio and certain data enhancements. 
Default or computed parameters may be used by the 
server entity (102) when no client-specified parameters 
are available. Upon receiving a request (1304) for a data 
file, the server entity (102) retrieves (1418) the fre- 
quency domain (908) form of the data file, quantizes 
(1604) frequency domain coefficients included in the 
frequency domain (908) form of the data file according 
to the parameters, compresses (1610) the quantized 
frequency domain coefficients into a compressed data 
file, and transfers (1 422) the compressed data file to the 
client entity (104, 106). 
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Description 



[0001] The present invention relates to the transfer of compressed files from a server computer to a client computer, 
preferably to a method and system for compressing stored data files by a server computer in a memory-efficient manner 
5 according to parameters that reflect the client computer's capabilities and the preferences of a user of the client com- 
puter. 

[0002] During the past five years, the volume of information exchanged between computer systems via the Internet 
has dramatically increased. Along with an increase in the volume of information, there has been a rapid increase in the 
variety of information exchanged via the Internet. Ten years ago, the Internet was used predominately for transfer of 

10 simple text files and larger binary data files. Today, users of personal computers ("PCs") routinely request and receive 
from server computers, via the Internet, complex graphical displays, high-resolution still images, recorded music and 
other audio files, animation and video files, and even live broadcast of relatively high-resolution video images. 
[0003] While the greatly increased volume and rate of data exchange over the Internet has fueled many technolog- 
ical and commercial advances, a number of server-related problems have been exacerbated, particularly with respect 

75 to the transfer of data files, such as images and audio files, that are large in size and that may be rendered for presen- 
tation on a number of different types of user rendering and presentation devices. The rate at which data can be trans- 
ferred from a server computer to a client computer via the Internet, and particularly to home PCs, may be quite limited. 
Files containing data for still images and video images, for example, may be quite large, on the order of hundreds of 
kilobytes to tens or hundreds of megabytes. A user interactively requesting and displaying images on a home PC from 

20 a server via the Internet may often encounter very long data transfer delays due to the large sizes of requested image, 
audio, and video files. A primary technique for improving data transfer rates is to compress the data prior to transferring 
it from a server computer to a client computer. Once the data has arrived at the client computer, the data can be decom- 
pressed to restore the image to a displayable format. Thus data compression provides a means for decreasing the 
amount of data that needs to be transferred in order to transfer an image from the server to the client computer more 

25 quickly. 

[0004] Many different types of data compression and decompression algorithms are available. Under lossless data 
compression and decompression algorithms, the stored image is identical, in information content, to the original image 
resident on the server computer. Under lossy data compression and decompression, the restored image may contain 
less information, and may also contain certain visual artifacts that arise during the compression and decompression 
30 processes. Generally, greater compression ratios can be achieved by lossy compression and decompression algo- 
rithms. In many cases, the information lost during lossy compression and decompression may be unnecessary, 
because the client computer is incapable of using the lost information for improving the rendering and presentation of 
the data. For example, a high-resolution image file that can be displayed on a high-end specialized graphics terminal 
may contain far greater information or, in other words, greater image detail, than can be displayed by a relatively low- 
35 resolution PC display screen. In addition, various mathematical manipulations that can be incorporated into the com- 
pression and decompression algorithms provide a means for altering data so that the data can be rendered more faith- 
fully by different types of rendering and presentation components. For example, the visual appearance of a color image 
rendered for display on a CRT screen may differ dramatically from the visual appearance of the same color image 
printed on a color printer. Compression and decompression algorithms can incorporate different types of enhancement 
40 algorithms in order to tailor a restored image for rendering and presentation on a particular device. Using the same 
example, if the server computer can determine that a user is requesting an image file in order to print the file on a 
printer, the server computer can compress the image in a way that will allow the image to be restored on the user's com- 
puter in a form that produces the visual appearance of the image as displayed on the CRT screen. Unfortunately, the 
rendering and presentation characteristics of various rendering and presentation components, including CRT and 
45 ■ active matrix display screens and various printing devices, may differ dramatically from one type of device to another. 
[0005] Server computer architects, Internet providers, and digital image processing scientists have recognized the 
desirability of serving differently compressed data files, including image files, to different users in order to provide the 
greatest possible data compression, and concomitant best possible data transfer rate, without unacceptable loss of 
information and to provide compressed data files that can be rendered and presented as faithfully as possible on differ- 
so ent types of user rendering and presentation devices. A common technique is to prepare, in advance, a number of dif- 
ferent compressed versions of each data file and to provide to a given user that version of a data file that most closely 
matches the capabilities of the user's computer and Internet connection and that most closely matches the user's pref- 
erence. However, storage space on server computers is limited. Even compressed data files take up a large amount of 
data storage space. Practically, only a limited number of compressed versions of each particular data file served by a 
55 server computer can be economically and conveniently stored. Thus, only a very crude, low-granularity matching of 
compressed images to user computer capabilities and user preferences can be achieved in this way. On-demand com- 
pression of uncompressed and unprocessed data files is computationally expensive, incurs excessively long transfer 
delays, and required excessive amounts of server computer memory. Server architects and Internet developers have 
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thus recognized a need for a method and system for tailoring data file compression more closely to user computer capa- 
bilities and user preferences, and for doing so in a more efficient manner with respect to server data storage and server 
memory resources. 

[0006] The present invention seeks to provide improved data transfer. 
5 [0007] According to an aspect of the present invention there is provided a method of providing a data file as spec- 
ified in claim 1 . 

[0008] According to another aspect of the present invention there is provided a data file transfer system as specified 
in claim 9. 

[0009] One embodiment of the present invention provides a method and system for on-demand data compression 
10 of data files for transfer from a server computer to a client computer. A single compressed, or partially compressed, ver- 
sion of the data file is stored on the server computer. A user may register various user preferences and user computer 
capabilities with the server computer prior to requesting the data file, or may indicate some or all of the preferences and 
capabilities at the time of the request for the data file. The server computer then compresses the requested data file 
according to the capabilities of the user's computer and user preferences, using default values for parameters not spec- 
15 if ied by the user. The preferences and capabilities-based compression is achieved in a particularly computationally and 
memory efficient manner. The compressed data file is then sent from the server computer to the user computer, where 
it is decompressed and rendered for presentation by the user's computer. 

[0010] In different embodiments of the present invention, different types of data files are compressed and decom- 
pressed using different compression and decompression algorithms. In one embodiment of the present invention, 

20 image data formatted according to the Joint Photographic Expert's Group ("JPEG") are compressed and decom- 
pressed according to JPEG compression and decompression algorithms, modified with various image enhancement 
techniques for efficient transfer of JPEG images from a server computer to a user's computer and for faithful rendering 
and presentation of the JPEG images by the user's computer. The method of the present invention may be applied to 
images formatted according to other standards, or to audio or video data files. Moreover, the technique of the present 

25 invention may be applied to the rendering and presentation of data files on various rendering and presentation devices 
directly connected to the computer system on which the data files are stored. 

[0011] An embodiment of the present invention is described below, by way of example only, with reference to the 
accompanying drawings, in which: 

30 Figure 1 illustrates the server/client environment in which the present invention can be employed. 

Figure 2 illustrates one conceptual approach to providing a JPEG image at different compressions and with differ- 
ent enhancements to different client computers. 
Figure 3 illustrates a representation of a two-dimensional pixel array. 
Figure 4 illustrates a common computational representation of a color image. 

35 Figure 5 illustrates an 8-pixel x 8-pixel subsection from the pixel array shown in Figure 3. 

Figure 6 illustrates a three-dimensional plot of frequency domain values on the u,v coordinate plane that might 
arise from and 8-pixel x 8-pixel spatial domain subimage. 

Figure 7 represents the magnitude of the frequency domain values, along a vertical axis, with relation to the dis- 
tance of the frequency domain value, along a horizontal axis, from the frequency domain origin. 
40 Figures 8-1 0 illustrate JPEG compression and decompression techniques in greater detail. 
Figure 1 1 A shows a one-dimensional frequency domain representation of an image. 
Figure 1 1 B illustrates the functional representation of a low pass filter. 

Figure 1 1 C shows the frequency domain representation of a printed version of the image represented in Figure 
11A. 

45 Figure 12A shows a representation of a high pass filter inverse to the low pass filter of Figure 1 1 B. 

Figure 1 2B shows a representation of the high pass filter of Figure 1 2A applied to the frequency domain represen- 
tation of the image shown in Figure 1 1 A. 

Figure 12C shows the frequency domain representation of a printed version of the transformed representation 
shown in Figure 12B. 

so Figure 1 3 is a high-level flow control diagram of an image serving process that runs on a server computer. 
Figure 14 is a flow control diagram of the routine "provide_image." 
Figure 15 is a flow control diagram of the routine "choose_Qc_and_Q^> ,, 
Figure 16 is a flow control diagram of the routine "compress." 

55 [0012] One embodiment of the present invention relates to the serving of JPEG images stored on server computers 
to user computers via the Internet. Each user may register a set of user computer capabilities and user preferences with 
the server computer and, in addition, a user may specify a set of user computer capabilities and user preferences while 
requesting a particular JPEG image from the server computer. Upon receiving a request for a JPEG image file from a 
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user computer, the server computer retrieves a compressed or partially compressed version of the JPEG image file and 
compresses it according to the specified capabilities of the requesting user's computer and according to any specified 
user preferences. This compression according to capabilities and preferences is carried out in a computationally effi- 
cient and memory-efficient manner. The JPEG image file compressed according to the capabilities of the user's com- 

5 puter and the user's preferences, is then transferred over the Internet to the user's computer. 

[0013] Figure 1 illustrates a server/dient environment in which an embodiment of the present invention is 
employed. A number of JPEG images are stored on a server computer 102. Client computers 104 and 106 are con- 
nected to the server computer 1 02 via the Internet 1 08. Client computer 1 06 and the server computer 1 02 are intercon- 
nected to the Internet 108 via high-speed/high-bandwidth connections 110 and 112, while client computer 104 is 

10 connected to the Internet via a relatively low-bandwidth modem and telephone line connection 114. Client computer 
106 is interconnected with a high-resolution specialized graphics display device 1 16 and a high-resolution color printer 
118. Client computer 104 is interconnected with a small, low-resolution display device 120 and a low-resolution printer 
122. In the case where users operating client computers 104 and 106 both request the same large JPEG image file, it 
would be desirable for the server computer 1 02 to compress the image file differently for transmission to the two users. 

15 In the case of the user of client computer 104, it would be desirable for the server computer 102 to provide a highly- 
compressed version of the JPEG image file to client computer 1 04 both in order to minimize the amount of data trans- 
ferred over the low-bandwidth connection 1 14 and to eliminate transmission of high -resolution detail that cannot be ren- 
dered and displayed on the low-resolution display device 120. In the case that the user of client computer 104 wishes 
to print the image on the low-resolution color printer 122, it would be desirable for the server computer 102 to provide 

20 to the client computer 1 04 a highly-compressed JPEG image file enhanced for a more faithful printing on the low-reso- 
lution color printer 122. On the other hand, in view of the relatively high bandwidth of connections 1 10 and 1 12, and the 
high resolution of the specialized graphics display device 116, it would be desirable for the server computer 1 02 to pro- 
vide the requested JPEG image file to client computer 106 in a less-compressed form with less information loss, since 
the high-resolution data can be rendered and displayed effectively on the specialized display device 1 1 6 and since the 

25 higher bandwidth of connections 1 1 0 and 1 1 2 can provide a sufficient rate of data transfer to quickly transfer a larger, 
less-compressed version of the JPEG image. If the user of client computer 106 intends to print the requested JPEG 
image on the high-resolution color printer 118, it would be desirable for the server computer 102 to enhance the com- 
pressed JPEG image file for faithful presentation on the high -resolution color printer 1 1 8. The enhancement desirable 
for the high-resolution color printer 118 may be quite different from the enhancement desirable for the low-resolution 

30 color printer 1 22. Moreover, it may be the case that JPEG images display, in general, more brightly on the low-resolution 
display device 120 than on the high-resolution display device 116. Therefore, it would be desirable for the server com- 
puter to compress the JPEG image in a way that enhances the brightness of the image when transferring the JPEG 
image to client computer 1 06. 

[001,4] Figure 2 illustrates one conceptual approach to providing a JPEG image at different compressions and with 

35 different enhancements to different client computers. In this approach, a given JPEG image can be compressed and 
enhanced in many different ways, and the resulting versions of the JPEG image can all be stored within the server com- 
puter. Continuing with the example presented in Figure 1 , the various versions of the JPEG image can be conceptual- 
ized as stored within a logical Cartesian volume 202 with orthogonal axes representing the compression ratio 204, a 
scaling factor 206, and an intensity factor 208. Thus, the compression ratios of the stored JPEG image versions 

40 increase downward along vertical columns 210-213 within the Cartesian volume. A scaling factor, corresponding to 
boosting of certain portions of frequency signal of the image, discussed further below, may increase along horizontal 
rows 214-218 of versions of the JPEG image within the Cartesian volume. Finally, the intensity, or brightness, of the 
image within the Cartesian volume may increase along rows orthogonal to the plane of Figure 2, such as the row indi- 
cated by arrow 220. Thus, when the server computer receives a request for the JPEG image, the server computer can 

45 select a particular pre-compressed version of a JPEG image within the Cartesian volume shown in Figure 2 having a 
compression ratio, scaling factor, and intensity that most closely matches a desired compression ratio, scaling factor, 
and intensity determined from the requesting computer's capabilities and requesting user's preferences. Returning to 
the example of Figure 1, when client computer 104 requests a particular JPEG image, the server computer 102 may 
select a compression ratio above which the visual appearance of the image as displayed on visual display device 120 

so would be deleteriously affected by the increased information loss that accompanies higher compression ratios, select 
either no intensity enhancement or a slight intensity enhancement, and select a scaling factor appropriate to enhance 
the JPEG image for faithful rendering and display on the low-resolution display monitor 120. The server computer may 
then select the version of the compressed JPEG image from the Cartesian volume 202 in Figure 2 closest to the point 
in space defined by the chosen compression ratio, scaling factor, and intensity factor. 

55 [0015] Unfortunately, the approach illustrated in Figure 2 has several major drawbacks. One drawback is that even 
compressed JPEG image files are relatively large. Although, in some cases, compression ratios approaching 100:1 
may be achieved, compression ratios on the order of 20:1 to 40:1 are more typical. If only ten different compression ratio 
levels, scaling factor levels, and intensity factor levels are used to define the Cartesian volume shown in Figure 2, 1 ,000 
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different compressed versions of each JPEG image file would need to be stored by the server. Considering the wide 
variety of image rendering and presentation devices, as well as the wide variety of capabilities of client computers and 
Internet connections to client computers, a much larger range of compression ratio values, scaling factors, and image 
factors may be desirable to achieve an effective level of compression tunability. Because data storage is limited, storing 
hundreds or thousands of different compressed versions of each JPEG image is an extremely expensive and inefficient 
solution. Moreover, a considerable amount of computer processing time would be devoted to preparing, in advance, all 
the different compressed versions of each JPEG image file. 

[0016] A second approach to providing tunable compressability for Internet servers is to store each image in 
uncompressed form, as one or more pixel arrays, and, upon receiving a request from a client computer for the image, 
compress the image, according to the JPEG compression and enhancement algorithms, using a compression ratio, a 
scaling factor, and an intensity factor chosen to meet the capabilities of the requesting client computer and the prefer- 
ences of the user operating the client computer. However, compression is a computationally intensive task, as will be 
discussed below, and complete on-demand compression from an uncompressed image to a desired compressed 
image for Internet transfer would introduce unacceptable transfer delays. Furthermore, the storing of uncompressed 
image files is again limited by server data storage capacity, and compression requires large amounts of contiguous 
blocks of server memory. 

[0017] To facilitate discussion of the present invention, Figures 3-12, and the discussion below related to Figures 3- 
12, describe computation representation of images and image compression. A monochrome image is commonly rep- 
resented in a computer as a two-dimensional pixel array. Figure 3 shows a representation of a two-dimensional pixel 
array. Each cell within the pixel array 302, such as cell 304, represents the smallest displayable unit within the image. 
Cells within the pixel array 302 are referenced via a two-dimensional Cartesian coordinate system 306. Thus, for exam- 
ple, cell 304 has Cartesian x, y coordinates of (0,7). Each cell within the pixel array 302 contains a value. For mono- 
chrome images, the value is a grayscale value. Images are often represented by devoting 8 bits for each pixel, resulting 
in grayscale values ranging between 0 and 255. Commonly, a grayscale value of 0 represents black, or the absence of 
light emanating from the pixel, and a grayscale value of 255 represents white. An image can be considered to be a two- 
variable function defined over the two-dimensional pixel array: 

grayscale value = f(x,y) 

30 For example, if pixel 304 in Figure 3 has a grayscale value of 1 00, then, in functional notation, 1 00 = f(0J). 

[0018] Color images are represented in a slightly more complex fashion. Figure 4 illustrates a common computa- 
tional representation of a color image. The image is represented by three different two-dimensional pixel arrays 402- 
404. The arrays share the same two-dimensional Cartesian coordinate system 406, with the pixels in all three two- 
dimensional pixel arrays aligned and in register. For example, pixel (1,1) 408 of pixel array 402 is aligned with, and in 

35 register with, pixel (1,1) 410 of pixel array 403 and pixel (1,1) 412 of pixel array 404. The values in all three pixels 408, 
41 0, and 412 are used to generate a color value displayed for pixel (1 ,1 ) in the displayed image. Different meanings may 
be assigned to the values stored in three pixel arrays that represent a color image. In one representation model, each 
plane is assigned to a different primary spectral component: red, green, and blue. The value stored in a pixel of the blue 
pixel array, for example, indicates the intensity of the blue primary spectral component displayed for the corresponding 

40 pixel in the displayed image. In the YIQ color model, the Y pixel array contains luminance values and the I and Q pixel 
arrays contain color information. In the HSI color model, the H pixel array contains values for hue, the S array plane con- 
tains values for saturation, and the I pixel array contains values for the intensity of the pixels. The latter two color models 
are particularly useful because the intensity information is stored separately, in a single pixel array, from the color infor- 
mation stored in the other two pixel arrays. The human visual system has higher-resolution sensitivity to intensity differ- 

as ences in an image than to color differences. Various image enhancement and sharpening techniques can be applied, 
in the YIQ and HSI color models, to the intensity pixel array alone, perceptibly sharpening the image, without introduc- 
ing coloring misregistration or artifacts. In general, the compression, decompression, and image enhancement tech- 
niques, to be discussed below, can be applied both to the single pixel array of a monochrome image, such as the 
monochrome pixel array illustrated in Figure 3, or can be separately applied to the three pixel arrays that together rep- 

50 resent a color image. Thus, in the following discussion, the compression, decompression, and enhancement tech- 
niques will be discussed without regard to whether the image to which the techniques are applied is a monochrome 
image or a color image. 

[0019] Figure 5 illustrates an 8-pixel x 8-pixel subimage from the pixel array shown in Figure 3. In Figure 5, the pixel 
values (grayscale, intensity, hue, or other such value, depending on the type of image representation) is shown by the 
55 height of the columns ascending vertically from the cells of the two-dimension Cartesian pixel plane. Figure 5 is equiv- 
alent to a three-dimensional depiction of the two-variable function: grayscale = f(x,y) . The values of this function over 
the pixel array represents the spatial domain of the image. In common images, the spatial domain may include repetitive 
features that are repeated at constant intervals, or periods, within the image. An example would be the spatial domain 



5 



10 
15 



20 



BNSOOCID: <EP 1 079329 A2 J _> 



EP 1 079 329 A2 



representation of an image of the grid-like pixel array shown in Figure 3. In the spatial domain representation of the 
image, the vertical lines repeat at a fixed vertical interval, or period, and horizontal lines also repeat within the image at 
a fixed horizontal interval, or period. 

[0020] A common mathematical transformation of the spatial domain representation of an image to a frequency 
5 domain representation of an image is accomplished by the discrete cosine transform, represented below functionally: 
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where 



«(«) = J| foru = 1,2 A/-1 



25 a(v)= ±for v= 1.2 A/ - 1 



N = number of pixels on the side of a square spatial domain pixel array 
x.y = pixel coordinates in the spatial domain 
30 u,v = pixel coordinates in the frequency domain 

[0021 ] The discrete cosine transform ("DCT") transforms spatial domain pixel values, referenced by x,y coordinates 
in the spatial domain, to a frequency domain value referenced by u,v coordinates in the frequency domain. Figure 6 
illustrates a three-dimensional plot of the magnitudes, or absolute values, of frequency domain values on the u,v coor- 
35 dinate plane that might arise from spatial domain values such as the spatial domain values represented in Figure 5. As 
is apparent from the DCT equation, shown above, all pixel values in the spatial domain, for example all 64 pixels within 
the 8-pixel x 8-pixel array shown in Figure 5, contribute to each frequency domain value. A special case is the 0,0 fre- 
quency domain value F| 0 0) which represents the average of the spatial domain values over the entire spatial domain. 
This can be easily seen by substituting the value 0 for variables 9 u" and V" in the above DCT equation: 



N-1 AM 



F (o,o) - ^/ L L f (*,y) 

x=0 y=0 

45 

The remaining values in the frequency domain, in aggregate, tend towards the value 0. Non-zero frequency domain val- 
ues indicate periodic features within the spatial domain. The u, v coordinates that reference the value are inversely 
related to the length of the period in the spatial domain. The frequency domain representation of the image, such as the 
frequency domain representation shown in Figure 6, can be transformed back to the spatial domain, such as the spatial 
so domain representation shown in Figure 5, using the reverse DCT: 



,„» = e ii «<«>«<*> ^ p^} ~ 

u=0 v=0 



[0022] JPEG compression methodologies make use of certain characteristics of the frequency domain values gen- 
erated by DCT transformation of an image. Figure 7 represents the magnitude of the frequency domain values, along 
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a vertical axis, with relation to the distance of the frequency domain value, along a horizontal axis, from the frequency 
domain origin at u,v coordinates 0,0. The value at 0,0 706 generally has the largest magnitude, and the magnitudes of 
the values fall of rapidly with increasing distance from the frequency domain origin. The larger magnitude frequency 
domain values clustered around the frequency domain origin contain the bulk of the information content of the fre- 

5 quency domain representation of the image. The lower magnitude frequency domain values at greater distances from 
the origin correspond to higher frequency, more closely spaced, periodicities within the spatial domain representation 
of the image. The essence of JPEG image compression is to first transform subimages within the image, via the DCT, 
into the frequency domain. The frequency domain values are generally expressed as 1 1-bit unsigned integers to repre- 
sent the full range of frequency domain values from 0 up to IF(o,o)'- Then, the frequency domain values are quantized 

io to redistribute the higher frequency domain values over a smaller range of possible values, and to set the smaller fre- 
quency domain values to 0. For example, quantization of the frequency domain values in Figure 7 might redistribute the 
higher magnitude values in the range 708 as lower values and eliminate, or set to 0, the lower magnitude values in the 
range 710. This quantization would result in retaining the frequency domain values within a distance of d 1 712 from the 
frequency domain origin. In this case, assuming that the frequency domain values are symmetrically distributed about 

,5 the origin, quantization may produce a thirty-fold reduction in the number of non-zero frequency domain values. The 
remaining, quantized frequency domain values are then encoded by difference and Huffman encoding with tailing 0 val- 
ues funcated. 

[0023] Figures 8-10 illustrate the JPEG compression and decompression techniques in greater detail. In the first 
step of image compression, as illustrated in Figure 8, an image pixel array, such as the pixel array shown in Figure 3, is 
po decomposed into 8-pixel x 8-pixel subimages. During compression, each 8-pixel x 8-pixel subimage is compressed sep- 
arately, and .during decompression, each 8-pixel x 8-pixel subimage is separately restored. The compression and 
decompression technique is illustrated for an 8-pixel x 8-pixel subimage in Figures 9 and 1 0. 

[0024] Figures 9 and 1 0 illustrate the various steps involved in compressing and decompressing an 8-pixel x 8-pixel 
subimage. The compression steps are indicated by downwardly directed arrows, such as downwardly directed arrow 

25 902. and the decompression steps are indicated by upwardly directed arrows, such as upwardly directed arrow 904. In 
Figure 9, the unsigned spatial domain values in the 8-pixel x 8-pixel subimage 902 are transformed into signed values 
in the spatial domain subimage 906 by subtracting the value of 2 n1 from each spatial domain value where n is the 
length, in pixels, of one side of the subimage, in this case, 8. The signed spatial domain values in subimage 906 are 
then transformed into frequency domain values in the frequency domain subimage 908 via the DCT In Figure 10, the 

30 frequency domain values in the frequency domain subimage 1002 (identical to the frequency domain subimage 908 in 
Figure 9) are quantized by dividing each frequency domain value by corresponding compression quantization matrix 
("Q c *) and rounding the result to the nearest integer in order to produce the quantized frequency domain subimage 
1 004. The quantized frequency domain values are then selected, in the zig zag order indicated in Figure 1 0, to produce 
a one-dimensional array of quantized frequency domain values 1006. The one-dimensional array of quantized fre- 

35 quency domain values are then encoded by difference in Huffman encoding to produce a compressed encoded bit 
stream representation 1008 of the quantized frequency domain values. 

[0025] Decompression starts with decoding the representation of the quantized frequency domain values 1008 to 
the n-dimensional array of quantized frequency domain values 1006. This is a form of lossless decompression. The 
one-dimensional array of quantized frequency domain values 1006 is then rearranged to produce the quantized fre- 

40 quency domain value subimage 1004. Approximations of the original unquanitzed frequency domain values are calcu- 
lated by multiplying each quantized frequency domain value in subimage 1004 by a corresponding value in a 
decompression quantization matrix ("Qo") to produce an approximation of the original frequency domain subimage 
1002. This approximation results because the rounding step in the corresponding compression stage results in loss of 
information that cannot be recovered. Thus, the frequency domain subimage 1 002 resulting from decompression of the 

45 quantized frequency domain subimage 1 004 is an approximation of the original frequency domain subimage 1 002 gen- 
erated during the compression cycle. In Figure 9, the frequency domain subimage 908 (1002 in Figure 10) is trans- 
formed by the reverse DCT to the assigned spatial domain subimage 906, which is then transformed by the addition of 
the value 2 n1 for each spatial domain value to produce the final restored spatial domain subimage 902. Because of the 
information loss in the rounding step, discussed above, the restored spatial domain subimage 902 is generally different 

50 from the original spatial domain subimage that was previously compressed and then decompressed. 

[0026] Figures 11A-C and 12A-C illustrate one approach to frequency domain image enhancement. In Figures 
1 1A-C and Figures 12A-C, the magnitude of frequency domain values was plotted on the vertical axis and the magni- 
tude of the distance of the value from the frequency domain origin is plotted on the horizontal axis. Thus, Figure 1 1 A is 
similar to Figure 7. Figure 11A shows the one-dimensional frequency domain representation of an image. The fre- 

55 quency domain representation features a large F (0 0) value 1 102 and various higher-frequency peaks 1104-1 106 and 
valleys 1107-1109. The higher-frequency peaks are inversely related to spatial domain periodicities, as discussed 
above. 

[0027] Figure 1 1 B illustrates the functional representation of a low pass frequency fitter. Lower frequencies, close 
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to the frequency domain origin 1110, are unaffected by the low pass filter. However, frequency domain values are 
increasingly diminished, or more completely filtered, with increasing frequency, or distance from the frequency domain 
origin 1110. Frequency domain values higher than frequency 1 1 1 2 are completely filtered by the low pass filter func- 
tionally represented in Figure 11B. Many image presentation devices, such as printers, act as low pass filters. Figure 

5 1 1C shows the frequency domain representation of a printed version of the image represented in Figure 11 A. Figure 
1 1 C is generated by applying the low pass filter function represented in Figure 1 1 B to the frequency domain represen- 
tation of the image in Figure 1 1 A. Note that the shape of the curve in Figure 1 1 C at low frequencies is similar in shape 
and magnitude to the curve in Figure 1 1A, but, at higher frequencies, much of the detail of the curve in Figure 1 1 A has 
been lost in Figure 1 1C. The higher frequency detail corresponds to higher resolution periodicities in the spatial domain 

w representation of the image. A spatial domain image that has undergone frequency domain filtering by a low pass filter 
appears softened and fuzzy with respect to the original unfiltered spatial domain image. 

[0028] Figure 12A shows a representation of a high pass frequency filter inverse to the low pass frequency filter of 
Figure 1 1 B. If the high pass frequency filter of Figure 12A were applied to the low pass frequency filter of Figure 1 1 B, 
the resulting function would be a flat horizontal line that, when applied to a frequency domain image representation, 

15 such as the representation in Figure 1 1 A, would produce no change in the shape of the frequency domain image rep- 
resentation curve. If it is known in advance that an image, such as the image represented in the frequency domain in 
Figure 1 1 A, presented on a presentation device that acts as a low pass filter, such as the low pass filter shown in Figure 
11B, an inverse filter, such as the filter shown in Figure 12A, can be applied to the frequency domain values of the 
image, as shown in Figure 11A, to produce an enhanced frequency domain image shown in Figure 12B. When the 

20 enhanced frequency domain image shown in Figure 12B is presented by the presentation device acting as the loss 
pass filter shown in Figure 1 1 B, the resulting image will have the frequency domain representation of Figure 12C, sim- 
ilar to the original frequency domain representation in Figure 11 A. Such enhancements can be incorporated into the 
compression and decompression techniques illustrated in Figures 9 and 10 by choosing modified Q c and Qr> For 
example, when no enhancement is desired, Q c is commonly equal to Qr> However, when a high frequency boost is 

25 desired, such as the high frequency boost achieved by application of the high frequency pass fitters shown in Figure 
1 2A, a different Q D that accomplishes the high frequency boost can be generated from Q c by multiplying Q c by a scal- 
ing factor. Additional, more complex enhancements to account for other types of characteristics of presentation devices 
can be incorporated either into the Q c or the Qr> A default Q 0 is assumed in accordance with the JPEG compression 
standard. The JPEG compression standard- allows for inclusion of specialized Q D s into compressed image files for 

30 effecting various enhancements and visual alterations upon restoration of the image through decompression and pres- 
entation. 

[0029] The above description of JPEG image compression, decompression, and enhancement sets the stage for a 
concise delineation of one embodiment of the present invention. In this embodiment, a server computer stores a single 
version of each JPEG mage file that the server computer makes available to client computers. The single version of the 

35 JPEG image file may be stored either in a compressed form or, alternatively, may be stored in the form of frequency 
domain values, call coefficients, that result from DCT transformation of a spatial domain representation of the image. In 
either case, the stored JPEG image representation has preferably not undergone quantization with concomitant loss of 
information. For example, if the JPEG image file is stored in compressed form, the compression generally omits the 
quantization step and relies chiefly on difference and Huffman encoding in order to decrease the size of the stored 

40 image. In alternative embodiments, the single version of the JPEG image file may be stored in a slightly quantized form. 
In the case that the image file is stored as a set of frequency domain coefficients, the frequency domain coefficients are 
either unquantized or only slightly quantized. Under this embodiment, a client can pre-register client computer capabil- 
ities and user preference parameters with the server computer or, alternatively, the client computer capability and user 
preference parameters may be furnished to the server computer at the time that the server computer receives from the 

45 client computer a request for the JPEG image file. The server computer then determines a Q c and, if necessary and 
acceptable, a Q D that produce a compression ratio and image enhancements that correspond to the client computer 
capabilities and user preferences. The server computer then quantizes the frequency domain coefficients using the 
determined Q c and then further compresses the quantized frequency domain coefficients by difference and Huffman 
encoding, according to the JPEG compression standard. If a specialized Q D has been chosen by the server computer, 

so that specialized Q D is included in the compressed JPEG image file. The compressed JPEG image file is then sent to 
the requesting client computer. 

[0030] Under this embodiment, only a single compressed version of each JPEG image file is stored on the server 
computer. Thus, the problems of storing multiple versions of each JPEG image file, described with reference to Figure 
2, are avoided. Furthermore, the on-demand compression carried out by the server starts with frequency domain coef- 
55 ficients rather than a spatial domain representation of the JPEG image. This technique avoids the computational over- 
head of application of the DCT to spatial domain image representations and avoids the need for the server computer to 
devote large, generally contiguous, blocks of memory for storing the spatial domain representation of the image prior to 
application of the DCT. Because many powerful image enhancement techniques can be carried out during the quanti- 
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zation step, the server computer, under the present embodiment, can tailor on demand compression not only to pro- 
duce compression ratios compatible with the transfer bandwidth and display capabilities of the client computer, but can 
also enhance the image for more faithful rendering and presentation on client computer presentation devices, such as 
color printers. 

5 [0031] Figures 13-16 provide a flow control diagram representation of one embodiment of the present invention. 
Figure 13 is a high-level flow control diagram of an image serving process that runs on a server computer. In step 1302, 
the image serving process waits for a next request from a client computer and, when the next request arrives, awakens 
and begins processing the request. In step 1304, the image serving process determines whether the request is a 
request for a JPEG image file. If so, then in step 1306, the image serving process calls the routine "providejmage" to 

io retrieve the image from a data storage component, such as a disk, and send the image to the requesting client compu- 
ter, after which the image serving routine returns to step 1302 to wait for the next client request. If the request was not 
for a JPEG image, as detected by the image server routine in step 1304, the image serving routine determines, in step 
1308, whether the client computer is requesting to register a set of client computer capabilities and user preferences to 
be used for future image requests. If so, then the image serving routine, in step 1310, calls a routine 

75 M register ..preferences" to collect the client computer capabilities and user preferences and store them in a database or 
other storage facility for later use in processing requests for images from the client computer. The routine 
" register jareferences" may be implemented in many different fashions depending on the storage and retrieval mecha- 
nisms desired, and will not be discussed further. Finally, if the request received from the client computer is not a request 
to register client computer capabilities and user preferences, then the request is handled in step 131 2 by calling the rou- 

20 tine "handlejrtherjequests" which will not be discussed further. Following completion of the calls to the routines 
"registe preferences" and "handle j3ther_requests," the image serving routine returns to step 1302 to wait for addi- 
tional requests from client computers. 

[0032] Figure 1 4 is a flow control diagram of the routine "providejmage." This routine is called by the image serving 
process in step 1306 in Figure 13. In step 1402, the routine "providejmage" determines whether the request for an 

25 image from a client computer includes an indication of client computer capabilities and user preference parameters and 
whether the request for the image includes a rule for applying the parameters. If so, then in step 1 404, the routine 
"providejmage" sets target parameters equal to the indicated parameter values and, if the rule has been indicated, 
sets the target rule to the indicated rule. An example of an indicated parameter might, for example, be a parameter indi- 
cating the bandwidth of the client computer's interconnection with the Internet or, as another example, a parameter indi- 

30 eating the resolution of the client computer's intended display device. A rule may, for example, indicate that no loss of 
image resolution can be tolerated by the client computer or, as another example, that less than optimal image resolution 
is preferable to any delay in the transfer of the image file incurred by less than optimal compression. In step 1406, the 
routine "provide jmage" determines whether any target parameters remain unspecified. If so, then in step 1408, the 
routine "providejmage" checks the contents of the database or another information storage mechanism to determine 

35 whether there are any client computer capabilities and user preferences stored for the client computer on the server. If 
there are stored parameters and rules, as detected in step 1410, then the server computer, in step 1412, sets any 
unspecified target parameters and the possibly unspecified target rule to the values stored in the database or other stor- 
age mechanism. If unspecified target parameters remain, as detected by the routine "providejmage" in step 1414, the 
routine "providejmage" sets any remaining unspecified target parameters, to default values in step 1416. 

ao [0033] In step 1418, the routine "providejmage" locates the requested image file within a data storage component 
of the server computer. In step 1420, the server computer determines whether the stored image file, which may be 
stored in a JPEG compressed form, is already sufficiently well-tailored to the client computer capabilities and user pref- 
erences determined in the preceding steps. If so, then the routine "providejmage" sends the retrieved file to the client 
computer in step 1422 and returns. Otherwise, the routine "providejmage" determines whether the file is stored as 

45 DCT coefficients or whether the file is stored as a compressed JPEG image file. In the latter case, the routine 
. "providejmage" decompresses the compressed JPEG file to DCT coefficients in step 1424, as described with refer- 
ence to Figure 10. In step 1426, the routine "providejmage" calls the routine "choose_Qo_and_Qo" to determine a 
suitable Q c and possibly a specialized Q D and then, in step 1428, uses the determined Q c to compress the DC coeffi- 
cients representing the image to a JPEG compressed image file in order to send the compressed JPEG image file, in 

so step 1 422, to the client computer. 

[0034] Figure 1 5 is a flow control diagram of the routine "choose_Q c _and_Q D ." The routine "choose_Q c _and_Q D " 
is called by the routine "providejmage" in step 1426 of Figure 14. In step 1502, the routine "choose_Q c _and_Q D " 
determines whether a target rule has already been set by the routine "providejmage." If not, then in step 1504, the rou- 
tine "choose_Qc_and_Qrj" determines whether a target rule is stored for the client computer in a database or other 

55 storage mechanism. If not, then the routine "choose_Q c _and_Q D ," in step 1506, selects a default rule. Otherwise, in 
step 1508, the routine "choose_Q c _and_Q D " selects the rule for the target rule that is stored in the database or other 
storage mechanism. At this point, the target parameters and target rule are fully specified, either from data supplied by 
the client computer, from default values, or from a combination of the two. In step 1510, the routine 
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"choose^Qc^and^QQ" determines whether it is possible to include a Q D in the compressed file sent to the client com- 
puter. If so, then instep 1512, the routine "choose_Q c _and_Q 0 B determines whether it is desirable to include a special- 
ized Q D in the compressed file for sake of image enhancement. If so, then in step 1514, the routine 
"choose_Q c _and_Q D " calculates Q c and Qq by using the target rule and target parameters. As discussed above, the 

5 target rule may specify various tradeoffs and absolute requirements that the server computer must make in balancing 
the target compression ratio and choosing various types of image enhancements. If inclusion of a specialized Q D is 
either not allowable or is not desirable in the case of the currently requested JPEG image file, the routine 
"choose_Qc_and_QD" calculates a Q c in step 1516 according to the target rule and target parameters. 
[0035] Figure 1 6 is a flow control diagram of the routine "compress." The routine "compress" is called by the routine 

io "provide jmage" in step 1428. In step 1602, the routine "compress" creates a new file that will contain the fully com- 
pressed JPEG image file that is to be sent to the client computer. In step 1604, the routine "compress" quantizes the 
DCT coefficients representing the requested JPEG image using the Oq determined by the routine 
"choose_Q c _and_Q D M in step 1606, the routine "compress" determines whether a specialized Q D is to be included in 
• the compressed JPEG image file, and, if so, includes the specialized Q D into the newly created file in step 1 608. Finally, 

15 in step 1610, the routine "compress" compresses the quantized DCT coefficients using Huffman indifference encoding, 
as discussed with reference to Figure 10, and places the resulting compressed quantized DCT coefficients into the 
JPEG image file. 

[0036] Although a particular embodiment has been described, the invention is not limited to this. Modifications will 
be apparent. For example, the method may be implemented as one or more software modules written in many different 

20 programming languages, using many different modular organizations, to run on many different hardware and operating 
system platforms. Although the embodiment described in detail related to serving JPEG images over the Internet, many 
different types of data can be transformed into the frequency domain for subsequent quantization to achieve compress- 
ibility and data enhancements, and many different communications vehicles may be be employed to transfer the com- 
pressed data. For example, video data, audio data, and image and graphics data can all be processed following 

25 transformation to the frequency domain. The transfer can be accomplished over intranets, serial communications net- 
works, telephone lines, and broadcast media when the client computer is remote from the server computer. In the case 
that a client process requests and receives compressed data from a server process running on the same computer sys- 
tem as the client process, various inter-process communications connections can be used, including mail boxes, shared 
memory, and sockets. Transformation of data to the frequency domain can be accomplished using many different math- 

30 ematical transforms implemented in many different ways, including Fourier transforms, fast Fourier transforms, Walsh 
transforms, Hadamard transforms, Haar transforms, Slant transforms, and Hotelling transforms. Moreover, many differ- 
ent types of compression techniques and algorithms may be used to compress quantized frequency domain coeffi- 
cients into compressed data files for transfer. In the above-described embodiment. JPEG compression techniques are 
used, but many other methodologies can be employed. 

35 [0037] The foregoing description, for purposes of explanation, used specific nomenclature to provide a thorough 
understanding. However, it will be apparent to one skilled in the art that the specific details are not required in order to 
practice the invention. 

[0038] The disclosures in United States patent application number 09/384,827, from which this application claims 
priority, and in the abstract accompanying this application are incorporated herein by reference. 

40 

Claims 

1. A method of providing a data file by a server entity to a user entity for presentation by the user entity, the method 
comprising: 

45 

storing a frequency domain (908) representation of the data file by the server entity (1 02); 
receiving by the server entity (102) a request (1304) from the user entity (104, 106) for the data file; 
when the data file is stored in a compressed form (1008), uncompressing (1424) the data file to a set of fre- 
quency domain coefficients; 

so quantizing and compressing (1428) the frequency domain coefficients according to a set of parameters to cre- 

ate a specially compressed data file; and 

sending (1 422) the specially compressed data file to the user entity (1 04, 1 06) where the specially compressed 
data file is subsequently decompressed and presented. 

55 2. A method as in claim 1 , wherein the server entity is a process running on a server computer (1 02), wherein the user 
entity is a process running on a user computer (104, 106), and wherein the server entity receives the request 
(1304) from the user entity via the Internet (108) and the server entity sends the specially compressed data file to 
the user entity via the Internet (108). 
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3. A method as in claim 1 or 2, wherein the date file contains data that can be rendered on a rendering device selected 
from: 

a visual display device (116, 120), in order to produce a visual display selected from: 

5 

a computer graphics display, 
a still image, 
a video image; 

w an audio broadcast device, in order to produce an audio signal; and 

a printing device (118, 122) in order to produce a printed visual image. 

4. A method as in claim 1 , 2 or 3, wherein the server entity stores the data file by transforming the data .file via a math- 
ematical transform to frequency domain coefficients and then stores the frequency domain coefficients, the math- 

15 ematical transform selected from a number of different mathematical transforms including a Discrete Cosine 
Transform and a Discrete Fourier Transform. 

5. A method as in any preceding claim, wherein the server entity stores the data file by transforming the data file via 
a mathematical transform to frequency domain coefficients, compresses the frequency domain coefficients into a 

20 compressed data file, and then stores the compressed data file, the mathematical transform selected from among 
a number of different mathematical transforms including a Discrete Cosine Transform and a Discrete Fourier Trans- 
form, the server entity compressing the frequency domain coefficients using difference and Huffman encoding. 

6. A method as in any preceding claim, wherein the parameters include a parameter indicating the bandwidth for 
25 transfer of the specially compressed data file, a parameter indicating the low pass frequency filter characteristics of 

a presentation device on which the user entity presents the data file, and a parameter indicating an intensity dimin- 
ishing characteristics of a presentation device on which the user entity presents the data file. 

7. A method as in any preceding claim, wherein quantizing and compressing (1428) the frequency domain coeffi- 
30 cients according to a set of parameters to create a specially compressed data file includes: 

choosing (151 6) a compression quantization matrix to achieve a compression ratio consistent with the param- 
eters; 

applying (1604) the compression quantization matrix to the frequency domain coefficients to quantize the fre- 
35 quency domain coefficients; and 

compressing (161 0) the quantized frequency domain coefficients to create a specially compressed data file. 

8. A method as in any one of claims 1 to 6, wherein quantizing and compressing (1428) the frequency domain coef- 
ficients according to a set of parameters to create a specially compressed data file includes: 

40 

choosing (1514) a compression quantization matrix to achieve a compression ratio consistent with the param- 
eters and to achieve an image enhancement consistent with the parameters; 

choosing (1514) a specialised decompression matrix to achieve an image enhancement consistent with the 
parameters; 

45 applying (1 604) the compression quantization matrix to the frequency domain coefficients to quantize the fre- 

quency domain coefficients; and 

compressing (1610) the quantized frequency domain coefficients to create a specially compressed data file, 
including in the specially compressed file the specialised decompression matrix. 

so 9. A data file transfer system for transferring a data file requested by a user for presentation by the user, the system 
comprising: 

a communications connection (1 14, 108, 110) between the user and a server (102); 

a server process running on the server (102) operable to receive a request (1304) for a data file from a user 
55 (104); 

a frequency domain (908) representation of the data file stored in a data storage component of the server (1 02) 
accessible to the server process; and 

an on-demand compression process operable to run of the server (102) for compressing (1428) the frequency 
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domain (908) representation of the data file into a specially compressed file for transfer via the communications 
connection to the user (104). 

10. A system as in claim 9, wherein, when the frequency domain (908) representation of the data file is stored in the 
5 data storage component in compressed form (1008), the server is operable to decompress (1424) the frequency 
domain (908) representation of the data file to frequency domain coefficients, wherein the on-demand compression 
process determines a target compression ratio, to quantize (1604) the frequency domain coefficients to achieve the 
determined target compression ratio, and to compress (1 610) the quantized frequency domain coefficients into the 
specially compressed data file, wherein the data file is selected from among an image file, a video file, an audio file, 
10 and a printed image file. 
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